Systems and methods to process electronic images to determine salient information in digital pathology

ABSTRACT

Systems and methods are disclosed for identifying a diagnostic feature of a digitized pathology image, including receiving one or more digitized images of a pathology specimen, and medical metadata comprising at least one of image metadata, specimen metadata, clinical information, and/or patient information, applying a machine learning model to predict a plurality of relevant diagnostic features based on medical metadata, the machine learning model having been developed using an archive of processed images and prospective patient data, and determining at least one relevant diagnostic feature of the relevant diagnostic features for output to a display.

RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Application No.63/021,955 filed May 8, 2020, the entire disclosure of which is herebyincorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure pertain generally toimage-based feature identification and related image processing methods.More specifically, particular embodiments of the present disclosurerelate to systems and methods for identifying diagnostic features basedon processing images of tissue specimens.

BACKGROUND

Pathology is a visual discipline that includes specializedinterpretation of morphological and histological patterns. Whole slideimages (WSI) of pathology specimens consist of hundreds of thousands ofpixels that a pathologist must review. Although not all of the pixelscontain relevant information, pathologists may need to review the entireWSI before rendering a diagnosis. The present disclosure describesvisualizations that allow pathologists to focus their attention onrelevant region(s) for a quick, complete, and correct diagnosis.

According to one or more embodiments in the present disclosure, outputsmay be leveraged from systems developed to identify specific features onwhole slide images of pathology tissue, saving pathologists time bytargeting their attention to areas on the whole slide image that arerelevant for a specific question, or for the diagnosis.

Additionally, the present disclosure describes additional methods forvisualizing identified cancerous foci of interest on whole slide imagesof digitized pathology images (e.g., other than heatmaps over allidentified regions of interest).

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for identifying a diagnostic feature of adigitized pathology image.

A method for identifying a diagnostic feature of a digitized pathologyimage, the method including: receiving one or more digitized images of apathology specimen, and medical metadata comprising at least one ofimage metadata, specimen metadata, clinical information, and/or patientinformation; applying a machine learning model to predict a plurality ofrelevant diagnostic features based on medical metadata, the machinelearning model having been developed using an archive of processedimages and prospective patient data; and determining at least onerelevant diagnostic feature of the relevant diagnostic features foroutput to a display.

A system for identifying a diagnostic feature of a digitized pathologyimage includes a memory storing instructions; and at least one processorexecuting the instructions to perform a process including receiving oneor more digitized images of a pathology specimen, and medical metadatacomprising at least one of image metadata, specimen metadata, clinicalinformation, and/or patient information; applying a machine learningmodel to predict a plurality of relevant diagnostic features based onmedical metadata, the machine learning model having been developed usingan archive of processed images and prospective patient data; anddetermining at least one relevant diagnostic feature of the relevantdiagnostic features for output to a display.

A non-transitory computer-readable medium storing instructions that,when executed by a processor, cause the processor to perform a methodfor identifying a diagnostic feature of a digitized pathology image, themethod including receiving one or more digitized images of a pathologyspecimen, and medical metadata comprising at least one of imagemetadata, specimen metadata, clinical information, and/or patientinformation; applying a machine learning model to predict a plurality ofrelevant diagnostic features based on medical metadata, the machinelearning model having been developed using an archive of processedimages and prospective patient data; and determining at least onerelevant diagnostic feature of the relevant diagnostic features foroutput to a display.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1A illustrates an exemplary block diagram of a system and networkfor identifying diagnostic features of an image, according to anexemplary embodiment of the present disclosure.

FIG. 1B illustrates an exemplary block diagram of the disease detectionplatform 100, according to an exemplary embodiment of the presentdisclosure.

FIG. 2 is a flowchart of an exemplary method for developing a featureidentification tool, according to an exemplary embodiment of the presentdisclosure.

FIG. 3 is a flowchart of an exemplary method for developing a featureidentification tool, according to an exemplary embodiment of the presentdisclosure.

FIG. 4 is a diagram illustrating an example crosshair output, accordingto an exemplary embodiment of the present disclosure.

FIG. 5 is a diagram illustrating an example output of a Field of View ofinterest output, according to an exemplary embodiment of the presentdisclosure.

FIG. 6 depicts an example system that may execute techniques presentedherein.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

Identifying areas of interest is a time-intensive process, whichincludes visual interpretation by specialists. As the number ofpathologists decreases across the world, the volume of pathologicalspecimens for review are increasing, which causes physician burnout andmisdiagnoses.

The process of analyzing an entire WSI for all slides in a patient casemay be entirely manual, which is extremely time-consuming and errorprone. Regions of interest may include features that are a fraction ofthe entire tissue (e.g., micrometers in size). At academic medicalcenters, pathologists in training (e.g., fellows) will manually reviewpatient's cases in advance of the pathologist's review. During review,the fellows will mark areas of interest and pre-write a diagnosis forthe pathologist's final review and diagnosis. In this method,pathologists are drawn to specific parts of the cases based on thetrainee's initial assessment. If pathologists are unsure of the finaland/or differential diagnosis, they have the option to send the materialto a different pathologist for a second opinion. The referralpathologist may only be sent the representative slide(s) for thespecific question—in this scenario, the pathologist's attention isfocused to a specific question and foci.

The present disclosure uses artificial intelligence (AI) technology thatdetects features of interests (e.g., biomarkers, cancer, histological,etc.) that may be used for pathological diagnosis and treatmentdecisions. This may be done at the case, part, block levels, and/orslide levels. Data and predictions are aggregated and made availableinstantaneously via any user interface (e.g., through a digitalpathology viewing system, report, or laboratory information system,etc.).

FIG. 1A illustrates an exemplary block diagram of a system and networkfor identifying diagnostic features of an image, according to anexemplary embodiment of the present disclosure.

Specifically, FIG. 1A illustrates an electronic network 120 that may beconnected to servers at hospitals, laboratories, and/or doctors'offices, etc. For example, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and laboratoryinformation systems 125, etc., may each be connected to an electronicnetwork 120, such as the Internet, through one or more computers,servers, and/or handheld mobile devices. According to an exemplaryembodiment of the present application, the electronic network 120 mayalso be connected to server systems 110, which may include processingdevices that are configured to implement a disease detection platform100, which includes a feature identification tool 101 for identifyingdiagnostic features pertaining to digital pathology image(s), and usingmachine learning to identify the diagnostic features, according to anexemplary embodiment of the present disclosure. Exemplary machinelearning models may include, but are not limited to, any one or anycombination of Neural Networks, Convolutional neural networks, RandomForest, Logistic Regression, and/or Nearest Neighbor.

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125may create or otherwise obtain images of one or more patients' cytologyspecimen(s), oncology specimen(s), slide(s) of the cytology/oncologyspecimen(s), digitized images of the slide(s) of the cytology/oncologyspecimen(s), or any combination thereof. The physician servers 121,hospital servers 122, clinical trial servers 123, research lab servers124, and/or laboratory information systems 125 may also obtain anycombination of patient-specific information, such as age, medicalhistory, cancer treatment history, family history, past biopsy orcytology information, etc. The physician servers 121, hospital servers122, clinical trial servers 123, research lab servers 124, and/orlaboratory information systems 125 may transmit digitized slide imagesand/or patient-specific information to server systems 110 over theelectronic network 120. Server systems 110 may include one or morestorage devices 109 for storing images and data received from at leastone of the physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Server systems 110 may also include processing devices forprocessing images and data stored in the storage devices 109. Serversystems 110 may further include one or more machine learning tool(s) orcapabilities. For example, the processing devices may include a machinelearning tool for a disease detection platform 100, according to oneembodiment. Alternatively or in addition, the present disclosure (orportions of the system and methods of the present disclosure) may beperformed on a local processing device (e.g., a laptop).

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and laboratory information systems 125refer to systems used by pathologists for reviewing the images of theslides. In hospital settings, tissue type information may be stored in alaboratory information system 125.

FIG. 1B illustrates an exemplary block diagram of a disease detectionplatform 100 for identifying diagnostic features pertaining to digitalpathology image(s), using machine learning.

Specifically, FIG. 1B depicts components of the disease detectionplatform 100, according to one embodiment. For example, the diseasedetection platform 100 may include a feature identification tool 101, adata ingestion tool 102, a slide intake tool 103, a slide scanner 104, aslide manager 105, a storage 106, and a viewing application tool 108.

The feature identification tool 101, as described below, refers to aprocess and system for identifying diagnostic features pertaining todigital pathology image(s), and using machine learning to identify thediagnostic features, according to an exemplary embodiment.

The data ingestion tool 102 refers to a process and system forfacilitating a transfer of the digital pathology images to the varioustools, modules, components, and devices that are used for classifyingand processing the digital pathology images, according to an exemplaryembodiment.

The slide intake tool 103 refers to a process and system for scanningpathology images and converting them into a digital form, according toan exemplary embodiment. The slides may be scanned with slide scanner104, and the slide manager 105 may process the images on the slides intodigitized pathology images and store the digitized images in a storage,such as storage 106 and/or storage devices 109.

The viewing application tool 108 refers to a process and system forproviding a user (e.g., pathologist) with specimen property or imageproperty information pertaining to digital pathology image(s), accordingto an exemplary embodiment. The information may be provided throughvarious output interfaces (e.g., a screen, a monitor, a storage device,and/or a web browser, etc.).

The feature identification tool 101, and each of its components, maytransmit and/or receive digitized slide images and/or patientinformation to server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125 over an electronic network120. Further, server systems 110 may include storage devices for storingimages and data received from at least one of the feature identificationtool 101, the data ingestion tool 102, the slide intake tool 103, theslide scanner 104, the slide manager 105, and/or viewing applicationtool 108. Server systems 110 may also include processing devices forprocessing images and data stored in the storage devices. Server systems110 may further include one or more machine learning tool(s) orcapabilities, e.g., due to the processing devices. Alternatively or inaddition, the present disclosure (or portions of the system and methodsof the present disclosure) may be performed on a local processing device(e.g., a laptop).

Any of the above devices, tools, and modules may be located on a devicethat may be connected to an electronic network 120, such as the Internetor a cloud service provider, through one or more computers, servers,and/or handheld mobile devices.

FIG. 2 is a flowchart illustrating an exemplary method of developing atool for identifying a diagnostic feature of a digitized pathologyimage, according to an exemplary embodiment of the present disclosure.For example, an exemplary method 200 (e.g., steps 202 to 206) may beperformed by the feature identification tool 101 automatically or inresponse to a request from a user (e.g., pathologist, patient,oncologist, etc.).

Exemplary Feature Identification Tool Development: An exemplary method200 for developing a feature identification tool may include one or moreof the steps below. In step 202, the method may include receiving one ormore digitized images of a pathology specimen (e.g., histology), andmedical metadata comprising at least one of image metadata, specimenmetadata (e.g., specimen type, available parts, gross description,etc.), clinical information (e.g., diagnosis, biomarker information, labresults, etc.), and/or patient information (e.g., demographics, gender,etc.). The method may include developing a pipeline that archivesprocessed images and prospective patient data. Additionally, data may bestored into a digital storage device (e.g., hard drive, network drive,cloud storage, RAM, etc.). In step 204, the method may include applyinga machine learning model to predict a plurality of relevant diagnosticfeatures based on medical metadata, the machine learning model havingbeen developed using an archive of processed images and prospectivepatient data (e.g., tissue type, specimen type, stain type, pathologist,etc.). In step 206, the method may include determining at least onerelevant diagnostic feature of the relevant diagnostic features foroutput to a display. Prediction results may be converted into a visualoutput depending on a type of user (e.g., pathologist, patient,oncologist, etc.), and the results may be displayed in a format based onthe type of user and the use case (e.g., interactive, structured,templatized, static, etc.).

FIG. 3 is a flowchart illustrating an exemplary method of using a toolfor identifying a diagnostic feature of a digitized pathology image,according to an exemplary embodiment of the present disclosure. Forexample, an exemplary method 300 (e.g., steps 302 to 306) may beperformed by the feature identification tool 101 automatically or inresponse to a request from a user (e.g., pathologist, patient,oncologist, etc.).

Exemplary Feature Identification Tool Use: An exemplary method 300 forusing a feature identification tool may include one or more of the stepsbelow. In step 302, the method may include receiving one or moredigitized images of a pathology specimen (e.g., histology), related caseand patient information (e.g., specimen type, case and patient ID, partswithin case, gross description, etc.), and information from clinicalsystem (e.g., assigned pathologist, specimens available for tests, etc.)into a digital storage device (e.g., hard drive, network drive, cloudstorage, RAM, etc.). In step 304, predictions, recommendations, andother data may be transmitted to an electronic storage device, and auser (e.g., pathologist, oncologist, patient, etc.) may be informed thatfoci of interest are available. The pathologist may opt into reviewing avisualization or report. In step 306, a visualization of foci ofinterest may be displayed in the form of a crosshair (see FIG. 4 ) onone or more points of interest (with or without descriptors or othertools) and/or field of views (see FIG. 5 ) on one or more areas ofinterest (with or without descriptors or other tools). Other visualindicators may also be displayed, such as an outline of an area ofinterest, which may have an irregular, non-geometric or polygonal shape.The pathologist may interact with and edit the foci and/or view eachregion of interest in order of priority or various other types ofordering. For example, the display may be automatically modified so asto zoom in or otherwise indicate a first region of interest with thehighest probability of diagnostic relevance. Upon receiving anindication such as a click from the pathologist, the display may beautomatically modified to take the focus of the display to a secondregion of interest with the second highest probability of diagnosticrelevance, and so on. The outputs and visualized regions may be loggedas part of the case history within the clinical reporting system.

Exemplary Cancer Detection Tool Development: An exemplary method fordeveloping a cancer detection tool may include one or more of the stepsbelow. The method may include a step of receiving one or more digitizedimages of a pathology specimen (e.g., histology), related information(e.g., specimen type, available parts, gross description, etc.),clinical information (e.g., diagnosis), and/or patient information(e.g., demographics, gender, etc.). The method may include a step ofdeveloping a pipeline that archives processed images and prospectivepatient data. The method may include a step of storing data into adigital storage device (e.g., hard drive, network drive, cloud storage,RAM, etc.). The method may include a step of generating a binary outputthat indicates whether or not a target feature is present. The methodmay include a step of generating, if the feature is present (e.g.,cancer present), a probability for cancer on all points of the wholeslide image. The method may include a step of converting the predictionresults into a form that may be visualized for and interpreted by theuser (e.g., pathologist, patient, oncologist, etc.). Additionally, theresults may be displayed in various effective formats depending on theuser and use case (e.g., interactive, structured, templatized, static,etc.).

Exemplary Cancer Detection Tool Use: An exemplary method for using acancer detection tool may include one or more of the steps below. Themethod may include a step of receiving one or more digitized images of apathology specimen (e.g., histology), related case and patientinformation (e.g., specimen type, case and patient ID, parts withincase, gross description, etc.), and/or information from a clinicalsystem (e.g., assigned pathologist, specimens available for tests, etc.)into a digital storage device (e.g., hard drive, network drive, cloudstorage, RAM, etc.). The method may include a step of outputting thesystem's predictions, recommendations, and data to an electronic storagedevice. A user (e.g., pathologist, oncologist, patient, etc.) may bemade aware that foci of interest and/or regions of interest areavailable. A pathologist may opt to review the visualization and/orreport. Visualization of foci of interest may be in the form of: showingone location that indicates the region with the highest statisticallikelihood for harboring cancer; showing top N locations (e.g., based onuser's preference) that indicate the regions with the higheststatistical likelihood for harboring cancer; showing the location orlocations for the region with values around the decision boundary fordetermining if the feature is cancer or not (e.g., three points aboveand three points below); and/or showing predictions on each piece oftissue on the slide (e.g., individual lymph nodes). Visualizations maybe provided with descriptors (e.g., statistical likelihood, etc.) andother tools (e.g., edit, delete, move, etc.). The pathologist mayinteract with and edit the foci. The pathologist may be directed to eachregion of interest in order of priority or based on other types ofordering. The outputs and visualized regions may be logged as part ofthe case history within the clinical reporting system.

Exemplary Cellular Feature Tool Development: Rather than detecting asingle feature, e.g., cancer, one or more embodiments may be used topredict multiple cellular features from input imagery. An exemplarymethod for developing a cellular feature tool may include one or more ofthe steps below. The method may include a step of receiving one or moredigitized images of a pathology specimen (e.g., histology), relatedinformation (e.g., specimen type, available parts, gross description,etc.), clinical information (e.g., diagnosis), and/or patientinformation (e.g., demographics, gender, etc.). The method may include astep of developing a pipeline that archives processed images andprospective patient data. Data may be stored into a digital storagedevice (e.g., hard drive, network drive, cloud storage, RAM, etc.). Themethod may include a step of generating binary outputs that indicatewhether or not each target feature is present. The method may include astep of identifying, for each feature that is present, all relevantareas where each feature is present in the whole slide image. The methodmay include a step of computing an overall score for each feature thatmay be utilized in a report. The method may include a step of convertingthe prediction results into a form that may be visualized for andinterpreted by the user (e.g., pathologist, patient, oncologist, etc.).The results may be displayed in various effective formats depending onthe user and use case (e.g., interactive, structured, templatized,static, etc.).

Exemplary Cellular Feature Tool Use: An exemplary method for using acellular feature tool may include one or more of the steps below. Themethod may include a step of receiving one or more digitized images of apathology specimen (e.g., histology), related case and patientinformation (e.g., specimen type, case and patient ID, parts withincase, gross description, etc.), and/or information from clinical system(e.g., assigned pathologist, specimens available for tests, etc.) into adigital storage device (e.g., hard drive, network drive, cloud storage,RAM, etc.). The method may include a step of outputting the system'spredictions, recommendations, and data to an electronic storage device.A user (e.g., pathologist, oncologist, patient, etc.) may be made awarethat foci of interest and/or regions of interest are available. Apathologist may opt to review the visualization and/or report.Visualization of foci of interest may be in the form of: showing onelocation that contains the highest density of the feature of interest(e.g., mitoses, glandular/tubular differentiation, nuclear pleomorphism,basal cells, etc.) (users may select which features to show or hide);showing top N locations (e.g., based on user's preference) that indicatethe regions with the highest statistical likelihood for harboringcancer; and/or showing the location or locations for the region withvalues around the decision boundary for determining if the feature iscancer or not (e.g., three points above and three points below). Themethod may include a step of showing indicators for multiple features atonce or separately. Visualizations may be provided with descriptors(e.g., statistical likelihood, etc.) and other tools (e.g., edit,delete, move, etc.). The pathologist may interact with and edit thefoci. The pathologist may be directed to each region of interest inorder of priority or based on other types of ordering. The outputs andvisualized regions may be logged as part of the case history within theclinical reporting system.

Exemplary Cancer Grade Tool Development: An exemplary method fordeveloping a cancer grade tool may include one or more of the stepsbelow. In this embodiment, a method is described for directing a user'sattention to specific cancer grades in a whole slide image, if they arepresent. The method may include a step of receiving one or moredigitized images of a pathology specimen (e.g., histology), relatedinformation (e.g., specimen type, available parts, gross description,etc.), clinical information (e.g., diagnosis), and patient information(e.g., demographics, gender, etc.). The method may include a step ofdeveloping a pipeline that archives processed images and prospectivepatient data. Data may be stored into a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.). The method mayinclude a step of generating binary output that indicates whether or nota target feature is present. The method may include a step ofidentifying, if the feature is present (e.g., grade of cancer), allrelevant areas where each feature is present in the whole slide image.The method may include a step of computing an overall score for eachfeature that can be utilized in a report. The method may include a stepof converting the prediction results into a form that can be visualizedfor and interpreted by the user (e.g., pathologist, patient, oncologist,etc.). The results may be displayed in various effective formatsdepending on the user and use case (e.g., interactive, structured,templatized, static, etc.).

Exemplary Cancer Grade Tool Use: An exemplary method for using a cancergrade tool may include one or more of the steps below. The method mayinclude a step of receiving one or more digitized images of a pathologyspecimen (e.g., histology), related case and patient information (e.g.,specimen type, case and patient ID, parts within case, grossdescription, etc.), and information from clinical system (e.g., assignedpathologist, specimens available for tests, etc.) into a digital storagedevice (e.g., hard drive, network drive, cloud storage, RAM, etc.). Themethod may include a step of outputting the system's predictions,recommendations, and data to an electronic storage device. User (e.g.,pathologist, oncologist, patient, etc.) is made aware that foci ofinterest and/or regions of interest are available. A pathologist may optto review the visualization and/or report. Visualization of foci ofinterest may be in the form of: showing one location that contains thehighest statistical likelihood of representing a particular grade ofcancer (e.g., Gleason Grades 3, 4, 5 for prostate cancer, Grade 1, 2, 3for breast cancer, Grades 1, 2, 3, 4 for lung cancer, etc.); showing topN locations (e.g., based on user's preference) that indicate the regionswith the highest statistical likelihood for representing or harboringcancer grade; and/or showing the location or locations for the regionwith values around the decision boundary for determining if the featureis cancer or not (e.g., three points above and three points below). Themethod may include a step of showing indicators for multiple features atonce or separately. Visualizations may be provided with descriptors(e.g., statistical likelihood, etc.) and other tools (e.g., edit,delete, move, etc.). The pathologist may interact with and edit thefoci. The pathologist may be directed to each region of interest inorder of priority or based on other types of ordering. The outputs andvisualized regions may be logged as part of the case history within theclinical reporting system.

Exemplary Cancer Type Tool Development: An exemplary method fordeveloping a cancer type tool may include one or more of the stepsbelow. For some tissues, multiple forms of cancer may occur (e.g.,lobular and ductal breast cancer). According to one embodiment, a user'sattention may be drawn to a type of cancer present in the image. Themethod may include a step of receiving one or more digitized images of apathology specimen (e.g., histology), related information (e.g.,specimen type, available parts, gross description, etc.), clinicalinformation (e.g., diagnosis), and patient information (e.g.,demographics, gender, etc.). The method may include a step of developinga pipeline that archives processed images and prospective patient data.Data may be stored into a digital storage device (e.g., hard drive,network drive, cloud storage, RAM, etc.). The method may include a stepof generating binary output that indicates whether or not a targetfeature is present. The method may include a step of identifying, if afeature is present (e.g., subtype of cancer), all relevant areas whereeach feature is present in the whole slide image. The method may includea step of computing an overall score for each feature that can beutilized in a report. The method may include a step of converting theprediction results into a form that may be visualized for andinterpreted by the user (e.g., pathologist, patient, oncologist, etc.).The results may be displayed in various effective formats depending onthe user and use case (e.g., interactive, structured, templatized,static, etc.).

Exemplary Cancer Type Tool Use: An exemplary method for using a cancertype tool may include one or more of the steps below. The method mayinclude a step of receiving one or more digitized images of a pathologyspecimen (e.g., histology), related case and patient information (e.g.,specimen type, case and patient ID, parts within case, grossdescription, etc.), and information from clinical system (e.g., assignedpathologist, specimens available for tests, etc.) into a digital storagedevice (e.g., hard drive, network drive, cloud storage, RAM, etc.). Themethod may include a step of outputting the system's predictions,recommendations, and data to an electronic storage device. A user (e.g.,pathologist, oncologist, patient, etc.) may be made aware that foci ofinterest and/or regions of interest are available. A pathologist may optto review the visualization and/or report. Visualization of foci ofinterest may be in the form of: showing one location that contains thehighest statistical likelihood of representing the subtype of cancer(e.g., ductal lobular breast cancer, melanoma for skin cancer, etc.);showing top N locations (e.g., based on user's preference) that indicatethe regions with the highest statistical likelihood for representing orharboring cancer subtype; showing the location or locations for theregion with values around the decision boundary for determining if thefeature is the cancer subtype or not (e.g., three points above and threepoints below). The method may include a step of showing indicators formultiple features at once or separately. Visualizations may be providedwith descriptors (e.g., statistical likelihood, etc.) and other tools(e.g., edit, delete, move, etc.). The pathologist may interact with andedit the foci. The pathologist may be directed to each region ofinterest in order of priority or based on other types of ordering. Theoutputs and visualized regions may be logged as part of the case historywithin the clinical reporting system.

Exemplary Non-Cancerous Feature Tool Development: An exemplary methodfor developing a non-cancerous feature tool may include one or more ofthe steps below. According to one embodiment, a method includesidentifying other non-cancer features, e.g., calcifications in breasttissue or identifying muscularis propria in bladder tissue samples. Themethod may include a step of receiving one or more digitized images of apathology specimen (e.g., histology), related information (e.g.,specimen type, available parts, gross description, etc.), clinicalinformation (e.g., diagnosis), and patient information (e.g.,demographics, gender, etc.). The method may include a step of developinga pipeline that archives processed images and prospective patient data.Data may be stored into a digital storage device (e.g., hard drive,network drive, cloud storage, RAM, etc.). The method may include a stepof generating binary output that indicates whether or not a targetfeature is present. The method may include a step of identifying, if thefeature is present (e.g., non-cancerous but suspicious features), allrelevant areas where each feature is present in the whole slide image.The method may include a step of computing an overall score for eachfeature that may be utilized in a report. The method may include a stepof converting the prediction results into a form that may be visualizedfor and interpreted by the user (e.g., pathologist, patient, oncologist,etc.). The results may be displayed in various effective formatsdepending on the user and use case (e.g., interactive, structured,templatized, static, etc.).

Exemplary Non-Cancerous Feature Tool Use: An exemplary method for usinga non-cancerous feature tool may include one or more of the steps below.The method may include a step of receiving one or more digitized imagesof a pathology specimen (e.g., histology), related case and patientinformation (e.g., specimen type, case and patient ID, parts withincase, gross description, etc.), and information from clinical system(e.g., assigned pathologist, specimens available for tests, etc.) into adigital storage device (e.g., hard drive, network drive, cloud storage,RAM, etc.). The method may include a step of outputting the system'spredictions, recommendations, and data to an electronic storage device.A user (e.g., pathologist, oncologist, patient, etc.) may be made awarethat foci of interest and/or regions of interest are available. Apathologist may opt to review the visualization and/or report.Visualization of foci of interest may be in the form of: showing onelocation that contains the highest statistical likelihood ofrepresenting a particular grade of cancer (e.g., fungus in derm samples,bacteria in colon samples, etc.); showing top N locations (e.g., basedon user's preference) that indicate the regions with the higheststatistical likelihood for representing or harboring clinicalpathological features; and/or showing the location or locations for theregion with values around the decision boundary for determining if thefeature is suspicious or not (e.g., three points above and three pointsbelow). The method may include a step of showing indicators for multiplefeatures at once or separately. Visualizations may be provided withdescriptors (e.g., statistical likelihood, etc.) and other tools (e.g.,edit, delete, move, etc.). The pathologist may interact with and editthe foci. The pathologist may be directed to each region of interest inorder of priority or based on other types of ordering. The outputs andvisualized regions may be logged as part of the case history within theclinical reporting system.

Exemplary Invasion Tool Development: In cancer pathology, one of thetasks of a pathologist is determining if invasion is present. Anexemplary method for developing an invasion tool may include one or moreof the steps below. The method may include a step of receiving one ormore digitized images of a pathology specimen (e.g., histology), relatedinformation (e.g., specimen type, available parts, gross description,etc.), clinical information (e.g., diagnosis), and patient information(e.g., demographics, gender, etc.). The method may include a step ofdeveloping a pipeline that archives processed images and prospectivepatient data. Data may be stored into a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.). The method mayinclude a step of generating binary output that indicates whether or nota target feature is present. The method may include a step ofidentifying, if the feature is present (e.g., invasion of cancer), allrelevant areas where each feature is present in the whole slide image.The method may include a step of computing an overall score for eachfeature that may be utilized in a report. The method may include a stepof converting the prediction results into a form that may be visualizedfor and interpreted by the user (e.g., pathologist, patient, oncologist,etc.). The results may be displayed in various effective formatsdepending on the user and use case (e.g., interactive, structured,templatized, static, etc.).

Exemplary Invasion Tool use: An exemplary method for using an invasiontool may include one or more of the steps below. The method may includea step of receiving one or more digitized images of a pathology specimen(e.g., histology), related case and patient information (e.g., specimentype, case and patient ID, parts within case, gross description, etc.),and information from clinical system (e.g., assigned pathologist,specimens available for tests, etc.) into a digital storage device(e.g., hard drive, network drive, cloud storage, RAM, etc.). The methodmay include a step of outputting the system's predictions,recommendations, and data to an electronic storage device. A user (e.g.,pathologist, oncologist, patient, etc.) may be made aware that foci ofinterest and/or regions of interest are available. A pathologist may optto review the visualization and/or report. Visualization of foci ofinterest may be in the form of: showing one location that contains thehighest statistical likelihood of representing evidence of invasivecancer (e.g., microinvasion in breast cancer, muscularis propriainvasion in bladder cancer, perineural invasion in prostate cancer,etc.); showing top N locations (e.g., based on user's preference) thatindicate the regions with the highest statistical likelihood forrepresenting or harboring evidence of cancer invasion; and/or showingthe location or locations for the region with values around the decisionboundary for determining if the feature is invasive or not (e.g., threepoints above and three points below). The method may include a step ofshowing indicators for multiple features at once or separately.Visualizations may be provided with descriptors (e.g., statisticallikelihood, etc.) and other tools (e.g., edit, delete, move, etc.). Thepathologist may interact with and edit the foci. The pathologist may bedirected to each region of interest in order of priority or based onother types of ordering. The outputs and visualized regions may belogged as part of the case history within the clinical reporting system.

According to one or more embodiments, a limited number of regions orfield of views on a whole slide image may be displayed to thepathologist and those selected regions may be sufficient to complete aspecific task in the diagnostic process (e.g., cancer detection,grading, triaging, etc.).

One or more embodiments may be implemented within a clinical workflow atthe hospital, lab, medical center as (1) Web application (cloud-based oron-premises); (2) Mobile application; (3) Interactive report; (4) Staticreport; and/or (5) Dashboard.

To improve ease of use, one or more embodiments may be implemented suchthat the area(s) with salient information may be organized into a reportwith overview information, or an interactive review/edit may befacilitated by the pathologist during review of the whole slide image.

One or more embodiments may be implemented such that multiple featuresmay be visualized on a single whole slide image.

The technical workflow according to one or more embodiments may be asfollows: a digitized whole slide image may be created and some or allmetadata may be available from hospital and hardware databases; imageand corresponding data may be passed into an artificial intelligence(AI)-based system and outputs may be generated; and/or some of theoutputs may be fed into a system that generates and displays thevisualization (e.g., one or multiple points or regions) to thepathologist based on the query of interest (e.g., cancer, nuclearfeatures, cell count, etc.).

Additionally, one or more embodiments of the present disclosure may beused for pre-screening (i.e., before a pathologist reviews an image)and/or after a diagnosis has been rendered (e.g., quality assurance).

As shown in FIG. 6 , device 600 may include a central processing unit(CPU) 620. CPU 620 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 620 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 620 may be connectedto a data communication infrastructure 610, for example, a bus, messagequeue, network, or multi-core message-passing scheme.

Device 600 also may include a main memory 640, for example, randomaccess memory (RAM), and also may include a secondary memory 630.Secondary memory 630, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage unit may comprise a floppy disk, magnetic tape, optical disk,etc., which is read by and written to by the removable storage drive. Aswill be appreciated by persons skilled in the relevant art, such aremovable storage unit generally includes a computer usable storagemedium having stored therein computer software and/or data.

In alternative implementations, secondary memory 630 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 600. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 600.

Device 600 also may include a communications interface (“COM”) 660.Communications interface 660 allows software and data to be transferredbetween device 600 and external devices. Communications interface 660may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 660 may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 660. Thesesignals may be provided to communications interface 660 via acommunications path of device 600, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 600 alsomay include input and output ports 650 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware.

The tools, modules, and functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. It isintended that the specification and examples be considered as exemplaryonly.

What is claimed is:
 1. A computer-implemented method for identifying a diagnostic feature of a digitized pathology image, the method comprising: receiving one or more digitized images of a pathology specimen, and medical metadata comprising image metadata, specimen metadata, clinical information, and patient information; applying a machine learning model to generate one or more predictions based on a presence of one or more pathological conditions in the one or more digitized images, the one or more predictions comprising a binary output to indicate a plurality of relevant diagnostic features based on medical metadata, the machine learning model having been developed using an archive of processed images and prospective patient data comprising at least one of a tissue type, a specimen type, and a stain type; generating, by the machine learning model, at least one relevant diagnostic feature of the relevant diagnostic features for output to a display, the at least one relevant diagnostic feature being based on the presence of a region having the one or more pathological conditions beyond a statistical likelihood, the pathological conditions comprising biomarkers, cancer, grade of cancer, non-cancerous features, nuclear features, and/or cell count; and providing, by the machine learning model, the at least one relevant diagnostic feature to the display as a region of interest indicated by at least one of an outline comprising a non-geometric shape.
 2. The computer-implemented method of claim 1, wherein the generating at least one relevant diagnostic feature further comprises: determining a probability of diagnostic relevance for each of the plurality of relevant diagnostic features; and determining a highest probability among the plurality of relevant diagnostic features for output to the display.
 3. The computer-implemented method of claim 1, wherein the generating at least one relevant diagnostic feature further comprises: determining at least one probability of diagnostic relevance that exceeds a predetermined value among the plurality of relevant diagnostic features for output to the display.
 4. The computer-implemented method of claim 1, wherein the generating at least one relevant diagnostic feature further comprises: determining a predetermined number of highest probabilities of diagnostic relevance among the plurality of relevant diagnostic features for output to the display.
 5. The computer-implemented method of claim 1, further comprising: determining a ranking of probabilities of diagnostic relevance of each of the plurality of relevant diagnostic features; automatically focusing the display on each of the relevant diagnostic features in order based upon the determining of the ranking of probabilities of diagnostic relevance; and receiving a user input to modify a focus of the display from a first relevant diagnostic feature to a second relevant diagnostic feature.
 6. The computer-implemented method of claim 1, wherein at least one field of interest is indicated on a digitized pathology image.
 7. The computer-implemented method of claim 1, wherein the method further comprises storing a collection of data into a digital storage device.
 8. The computer-implemented method of claim 1, wherein the method further comprises generating a probability for biomarkers, cancer, and/or histological features on all points of a whole slide image.
 9. The computer-implemented method of claim 1, wherein the method further comprises identifying a set of relevant areas where the one or more pathological conditions is present in a whole slide image.
 10. The computer-implemented method of claim 9, wherein the method further comprises computing an overall score for each pathological condition.
 11. A system for identifying a diagnostic feature of a digitized pathology image, the system comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to perform operations comprising: receiving one or more digitized images of a pathology specimen, and medical metadata comprising at least one of image metadata, specimen metadata, clinical information and/or patient information; applying a machine learning model to generate one or more predictions based on a presence of one or more pathological conditions in the one or more digitized images, the one or more predictions comprising a binary output to indicate a plurality of relevant diagnostic features based on medical metadata, the machine learning model having been developed using an archive of processed images and prospective patient data; generating, by the machine learning model, at least one relevant diagnostic feature of the relevant diagnostic features for output to a display, the at least one relevant diagnostic feature being based on the presence of a region having the one or more pathological conditions beyond a statistical likelihood, the pathological conditions comprising biomarkers, cancer, grade of cancer, non-cancerous features, nuclear features, and/or cell count; and providing, by the machine learning model, the at least one relevant diagnostic feature to the display as a region of interest indicated by at least one of an outline comprising a non-geometric shape.
 12. The system of claim 11, wherein the generating at least one relevant diagnostic feature further comprises: determining a probability of diagnostic relevance for each of the plurality of relevant diagnostic features; and determining a highest probability among the plurality of relevant diagnostic features for output to the display.
 13. The system of claim 11, wherein the generating of at least one relevant diagnostic feature further comprises: determining at least one probability of diagnostic relevance that exceeds a predetermined value among the plurality of relevant diagnostic features for output to the display.
 14. The system of claim 11, wherein the generating at least one relevant feature further comprises: determining a predetermined number of highest probabilities of diagnostic relevance among the plurality of relevant diagnostic features for output to the display.
 15. The system of claim 11, further comprising: determining a ranking of probabilities of diagnostic relevance of each of the plurality of relevant diagnostic features; automatically focusing the display on each of the relevant diagnostic features in order based upon the determining of the ranking of probabilities of diagnostic relevance; and receiving a user input to modify a focus of the display from a first relevant diagnostic feature to a second relevant diagnostic feature.
 16. The system of claim 11, wherein at least one field of interest is indicated on a digitized pathology image.
 17. A non-transitory computer-readable medium storing instructions that, when executed by a processor to perform a method for identifying a diagnostic feature of a digitized pathology image, the method comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to perform operations comprising: receiving one or more digitized images of a pathology specimen, and medical metadata comprising at least one of image metadata, specimen metadata, clinical information and/or patient information; applying a machine learning model to generate one or more predictions based on a presence of one or more pathological conditions in the one or more digitized images, the one or more predictions comprising a binary output to indicate a plurality of relevant diagnostic features based on medical metadata, the machine learning model having been developed using an archive of processed images and prospective patient data; generating by the machine learning model, at least one relevant diagnostic feature of the relevant diagnostic features for output to a display, the at least one relevant diagnostic feature being based on the presence of a region having the one or more pathological conditions beyond a statistical likelihood, the pathological conditions comprising biomarkers, cancer, grade of cancer, non-cancerous features, nuclear features, and/or cell count; and providing, by the machine learning model, the at least one relevant diagnostic feature to the display as a region of interest indicated by at least one of an outline comprising a non-geometric shape. 